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Preface 



This document represents an effort on the part of The Educational 
Systems Division of Scientific Management Associates to pull together in 
one document the evaluation materials and the instructional content 
pertinent to an understanding of the IVD process for 1972-73. 

Section I is an analysis of the on-site experience of the validators 
in the employment of the Instrument. Section II is an SMA evaluation of 
the ten national training workshops for validators, state and local 
project personnel. Section III includes our recommendations for Year II 
developmental activities based on input from the validators* critiques, 
the participant evaluation of the workshops, and SMA's involvement in 
the Instrument design phase. The three appendices complete the picture 
relative to additional data placed in the hands of workshop participants 
and selected BESE personnel. 

Ou»^ si nearest appreciation is directed to Title III, ESEA personnel 
for the opportunity of participating in this first IVD effort. 



\ 0. Robert Hanson 



Gloucester, New Jersey 
May 23, 1973 



Part I A Cortent Analysis of the Validator Self Analysis Forms 



A content and inter-item analysis of the Validator's Self Analysis 
(on-s'ite) Form' indicate: positive steps for the improvement of the I.V*D. 
procedure, and for the extensive and much needed revision of the validation 
instrument itself. 

There were 299 Self Analysis forms returned representing a clear 
majority of those serving on teams. An exact count is not possible for 
we do not have figures on validators serving on more than one team^ 
nor on the total number of on-site visits completed. It is our opinion, 
however, that the number of responses provides a thoroughly sufficient 
basis upon which to make generalizations. 

Of those responding, 292 or 97% felt their involvement on the team 
reflected their area of expertise. Similarly 294 or 98% indicated that 
the paint values they had assigned to their section was acceptable to 
their teammates. Further, the great majority (98%) indicated that there 
was both good team interaction in responding to the Instrument's questions, 
and to relations with state and project personnel • 

One may conclude from these findings that the validator selection 
procedure, and the trainer's instructions for team interaction were 
generally productive, and can be recommended for continuation in Year II 
development activities. 

Of thos.?; responding, 124 or 41% found the task of assessing the data 
and v/eighing responses; difficult, whereas 173 or 57% indicated either 



1 Please see next page. 
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Validator Self Analysis Form 



Please fill in the following information at the conclusion of the on-site 
validation and mail in the stamped envelope attached. 

Name Date 



Project Reviewed for Validation 



City State 

Your Address 



City State 

Section Reviewed 



1. Do you feel your involvement reflected your area of expertise? 
Yes No 

2. Were the point values for the section you validated generally acceptable to 
your teammates? 

Yes No 

3- Did you find the task of assessing tne data and weighing the responses 
difficult? 

Yes No 

Comments: 



4. Were there questions in your section that you found particularly difficult 

to answer in terms of assigning number weights? Yes No 

If "Yes", please indicate section and question numbers: 
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Do you feel that there was adequate team interaction and discussion in 
reaching a conclusion on each of the four sections of the Report? 
Yes No 

Please comment on areas of difficulty with respect to both the validation 
instrument, and the team's interaction with one another and with project 
personnel . 



2. 



little or no difficulty (or "difficult but doable") in responding.^ 
Similarly 131 or 43% indicated they experiericed difficulty in responding 
to questions in their assigned sections. For purposes of comparison^ 
'the responses to this question by class were as follows: 

Question #3 "Did you find the task of assessing the 

data and weighing the responses difficult?" 

Yei % No, 1 

Innovativeness 

N=71 28 39 43 61 

Effectiveness/Success 

N=66 31 46 35 53 

Cost Effectiveness 

N=76 31 41 45 59 

Exportuoility 

N=69 31 45 38 55 

These responses overall indicate that 43% of the respondents had 
difficulty in answering th? questions in their assigned section. The 
figures show a seven point response range for all classes, and a five 
point response range for those responding to Sections II, III and IV. 

Answering the questions required a two step procedure including 
1) assessing the adequacy, appropriateness and accuracy of the data 
(documentation and testimony), and 2) assigning number weights. Since 
this is a two part question, it is not possible to identify the greater 
area of difficulty in responding. It is our contention, however, that 
since the tv;o judgments are inextricably related that the respondent 
feedback indicates the need for major Instrument revision responsive 
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^ In Year II it would be appropriate to compare validators' responses on 
the basis of projects they worked on that were approved or disapproved. 
Correlations of responses v/ould assist in identifying where non-validated 
projects fall down in the documentation proceedings. 
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to the problematic questions identified in response to Question 4 (of the 
Self Analysis Form). 

Responses to Question #4 were as follows: 

"Were there questions in your section that you found particularly 
difficult to answer in terms of assigning number weights?" 

Yes. % No. i 

Innovativeness 

N=71 13 18 57 80 

Effectiveness/Success 

N=66 35 53 30 45 

Cost Effectiveness 

N=76 42 55 33 43 

Exportability 

N=69 31 45 38 55 



Again, as v/ith responses to Question #3, 43^ indicated they had 
problems with particular questions. This time, however, the response 
range between the four classes was 37, and for respondents to Sections 11 » 
III and IV the range was 10. Clearly those serving as validators to 
Sections II and III experienced the greatest difficulty in responding. 

Validators assigned to the Innovativeness questions (Section I) 
experienced fewer difficulties in responding since the Section contained 
only two questions and required the admission of the data as presented. 

It is our recommendation that Section I be deleted as a self-contained 
area for validator response, and that the information be presented to the 
validation team as a "given", and for information purposes only. Such 
a revision places the total responsibility for project identification on 
the State as stipulated by the law. The validator's role then is one 
not of judging the accuracy and appropriateness of the state's selection. 



but rather of scrutinizing the evidence for exemplariness against the 
criteria for effectiveness, cost, and dissemination potential. 



Excluding responses on this Section also allows for a more equitable 
distribution of points for the critical sections (II, III and IV). This 
recommendation is also supported by those validators not serving as team 
chairmen and responsible for Section I, i.e., there wasn't enough 
meaningful work to do. 

The following chart shows percentage responses by classes for 
questions one through five: 

Question §^ - "Do you feel your involvement reflected your 
area of expertise?" 

Question #2 - "Were the point values for the section you validated 
generally acceptable to your teammates?" 

Question #3 - "Did you find the task of assessing the data and 
weighing the responses difficult?" 

Question #4 - "Were there questions in your section that you 
found particularly difficult to answer in terms 
of assigning number weights?" 

Question #5 - "Do you feel that there was adequate team interaction 
and discussion in reaching a conclusion on each of 
the four sections of the Report?" 



Percentage Responses to Self Analysis Fonn 



Ques. 1 Ques. 2 Ques. 3 Ques. 4 Ques. 



I Innovativeness 
N=71 



Yes 
No 



100% 
0 



98% 
2 



39% 
60 



18% 
82 



98% 
2 



II Effectiveness/Success 
N=66 

Yes 
No 



98 
2 



98 
2 



46 
54 



53 
47 



95 
5 



Ques. 1 Ques. 2 Ques. 3 



Ques. 4 



Ques. 



Ill Cost Effectiveness 
N=76 



Yes 
No 



94% 
6 



98% 
2 



40% 
60 



55% 
45 



98% 
2 



IV Exportability 
N=69 



Yes 
No 



100 
0 



100 
0 



44 
56 



44 
56 



97 
3 



Clearly the ireas of difficulty are those of assessing the adequacy, 
accuracy and appropriateness of the data, and assigning number weights. 



Sect-' on II Areas of Difficulty 

Specific areas of difficulty for validators responrf-'ng to Section II 
(Effectiveness/Success) v/ere as follows: 

• multi-faceted questions, i.e., ansv/ering questions with more 

than one reference point 

• the issue of "adequacy" and "appropriateness" of the documentation, 

i.e., what do these terms mean? 

• the questions of "validity" and "reliability" for project generated 

tests, i.e., specifying the statistical definitions utilized 

• the issue of "significance", i.e., the term being used in a non- 

statistical and statistical sense 

• the issue of poorly stated objectives; the on-site revision of 

objectives, the absence of objectives, or objectives with 
inadequate, non-existent performance levels, or performance 
levels too high (or too low) 

• the issue of erratic documentation 

• the inappropriateness of the number weighting systems* for questions 

of different magnitudes 

• the issue of documentation as its own reward, i.e., extensive 

documentation for low priority questions potentially ending up 
as more numerically significant than high value questions vn'th 
lesser amounts of data 
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• the issue of validator role confusion* e.g., "is the function 

validation or evaluation?" 

• the issue of inadequate evOuation des'^gns v/ell completed (with 

resultant high scores) versus a more adequate design possibly 
less satisfactorily completed 

• projects with incomplete data either because of mid-point 

- development, inadequate orientation to the Instrument's data 
needs, or generally inadequate or inconclusive documentation 

• the issue of hearing testimony through on-site interviews and 

building such evidence into responses 

Questions Causing Difficulty in Section II 

The questions causing Section II validators the greatest amount 
of difficulty are as follows and in this order of priority: 

1. Question 12 - How accurately were the data analyzed? 

2. Question 10 - How would you assess the accuracy of data 

processing, i.e., scoring data verification 
and editing, data organization and tabulation? 

3. Question 7 - Are the instruments used to measure the major 

objectives reliable for the purposes for which 
they were used? 

4. Question 6 - Are the instruments used to measure the major 

objectives valid for the purposes for which they 
were used? 

Question 9 - To what extent is the processircg data, i.e., 
scoring, data verification and editing, data 
organization, tabulation appropriate in scope 
and format to the kinds of analysis and 
summarization needed to determine effectiveness/ 
success? 

Question 14 - To what extent does the project evaluation contain 
acceptable evidence that the performance of the 
participants was significantly improved? 

5. Question 3 - Based upon your analysis of the baseline data, the 

characteristics of the learner, and the purposes 
of the project, what proportion of the expected 
performance levels {as indicated in the objectives) 
are realistic? 



ERLC 



7. 



5. Question 15 - On the basis of the objectives, i.e, , anticipated 

outcomes, does the evaluation eviderxe indicate 
that the project activities have effectively 
improved participant behavior at the stated 
expectancy levels? 

6. Question 8 - To what extent were personnel administering the 

instruments qualified to administer the instruments? 

Question 11 - How extensively were the collected data analyzed 
i.e., did the project staff use a wide range of 
appropriate descriptive, inferential, and causal 
comparative analysis techniques? 

Question 13 - To what extent are conclusions supported by data 
(evidence) collected? 

Section III Areas of Difficulty 

Specific areas of difficulty for validators responding to Section III 
(Cost Effectiveness) include many of those previously stated plus the 
general validator recognition that while the questions dealt with "costs" 
they did not deal with "benefits", and that the data collection forms 
v/ere totally inadequate. It is our recommendation that provision be made 
for both the complete revision of the data collection section^ and that 
questions be included providing for data allowing comparif,ons between 
costs and achievement. Such questions would necessitate the construction 
of a formula allowing for both the clear identification of Title III funds, 
and for a per pupil per instructional hour cost from non-Title III funds 
(i.e., district, state, and federal— excluding Title III). 

Questions Causing Difficulty in Section III 

Questions causing Section III validators the greatest difficulty 
are as follows and in this order: 

ERIC 
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1. Question 6 - Consider "effectiveness" the rating given on the 
project's ability to meet as the predetermined 
perfomance levels of the objectives. Consider 
"cost" as the increased cost from the current 
per pupil expenditure in the district for the 
maintenance of the project. On the grid below 
rate the project for effectiveness and cost: 
Check the box which best describes this project 
and enter the score in the space to the right: 
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2. Question 3 - Give the cost breakdowns by developmental cost, 

installation (start up) cost, and continuation cost. 

(a) Estimated developmental cost $ 

(b) Estimated start up or installation cost 

if a LEA is to replicate your project $ 

(c) Estimated continuation cost {excluding 
developmental and installation cost) $ 

How would you rate the accuracy of the development, 
installation, and continuation of the data presented? 
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2. Question 5 - What is the probability that by the end of the 

project, this operation can replace related 
current operation? 

3. Question 7 - In your opinion do the total results (practices/ 

benefits) of the project justify the costs? 



Our recommendations are that: 

1. The grid be reformulated utilizing a cost effectiveness 
formula 

2. That question "5" be jointly answered by the validators for 
cost and for exportabili ty 

3. That upon revision questions "6" and "7" be answered by the 
entire team, and 

4. That question "7" be revised from a "Yes-No" to multiple 
points on a continuum 

5. That question "5" be deleted, or that projects which are new 
to the district (e.g., early childhood) not be penalized 

6. That provision be made to indicate per pupil costs which are 
additional for the installation stage , and v/hich are additional 
for the continuation stage (if any) 



Section IV Areas of Difficulty 

Specific areas of difficulty for those responding to Section IV 
(Exportabili ty) are as follows: 

• the absence of directions for and responses to "comprehensive and 

accurate learner descriptions" 

• the absence of accurate descriptions of institutional variables 

• the gene»-al absence of documentation responsive to replication 

• the absence of Instrument directives vis a vis descriptions of 

"home" and "community" variables 

• the difficulty of responding to what in effect is hearsay type 

testimony on the documentation of "expected" and "unexpected" 
constraints 

• the impossibility of assessing for number weighting the "extent" 

of community support 

• the absence of data relative to the need for "specialized" staff 
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Questions Causing Difficulty in Section IV 

Questions causing the greatest amount of difficulty are as follows 
and in this order; 

1. Question 4 - To what extent does the project contain comprehensive 

and accurate descriptions of the characteristics of 
the learner that are critical to the successful 
replication of the practice? 

2. Question 5 - To what extent does the project, contain comprehensive 

and accurate descriptions of institutional variables 
(school administration, climate for change, philosophy, 
support) critical to the replication of the project? 

3. Question 7 - To what extent is the documentation of the project's 

results responsive to project replication? 

4. Question 6 - To what extent does the project contain comprehensive 

and accurate descriptions of community and home 
variables critical to the replication of the project? 

5. Question 9 - Does the extent of the project's requirement for 

specialized staff detract from the potential for 
adoption by other districts? 

6* Question 3 - What is the extent of support of lay citizens of 
the community for this project? 

7. Question 15 - To what extent does the project document the 

expected and unexpected constraints or problems 
met and solved? 

8. Question 10 - Does the cost for staff training detract from the 

potential for adoption by other districts? 

Our recommendations for the revision of Section IV are: 

1. that detailed directions be given State and project personnel 
for accurately describing the learners in question, and that 
definitions be provided for "institutional", "home" and "community" 
variables necessary for project understanding and replication 
(Questions 5 & 6) 

2. the provision of numerical or quantitative guidelines for data 
presentation for use in making judgments on "extent" particularly 
as it applies to the issue of community support (Question 3) 
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3. the provision of guidelines (such as personnel/task matrices) 

for use in assessing the need for "specialized staff" (Question 9) 

4. While the larger issue of what constitutes adequate documentation 
needs to be dealt with in terms of simulated case studies in the 
training workshops, some greater degree of attention will need to 
be focused on the Instrument itself in terms of the numerical 
revaluing of the questions pertinent to replication. This 
question (#7) can then be answered by the validator's analysis 

of the key questions identified for replication. This recom- 
mendation includes, in all probability, the need for both coding 
the questions critical to replication, and, of course, changing 
the values of the questions relative to the magnitude of their 
importance. 



General Problem Areas (Responses to Self Analysis Form Question 6) 

The problems are listed in the order of their importance to the 
validators, i.e., the most frequently cited problem first, and so on. 

1. inadequate documentation, i.e., local project personnel 
were not ready for the on-site visit, and/or the data did 
not exist 

2. the Instrument was too limited in scope for those many projects 
with more than a single practice to validate 

3. the Instrument was too dependent upon well stated objectives, 
but without reference to the meaningful ness of the objectives 

4. the entire validation process requires a research design 
methodology and is, therefore ^ unresponsive to certain types 
of affective and psychomotor-centered projects 

5. the various terms in the Instrument are inadequately defined 
or not defined at all 

6. the point values for the questions are inappropriate and do 
not reflect information critical to success or replication 

7. the condition of the objectives often necessitated rewrites 

or the "leading of the witness" to write objectives responsive 
to what was actually being accomplished 

8. the need for a clear distinction between the role of validator 
and the role of evaluator, i.e., some few questions demanded 
not the review of documentation but the assessment of the value 
of the particular practice or procedure 

9. the issue of project procedures or process objectives having 
been deleted when in many cases the project's contribution 
to innovativeness/exemplariness was the process utilized 
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10. the problems raised by the conversion table, i*e,, ineffective 
but well documented projects receiving adequate points for 
validation 

11. the inequitable number weighting of the questions 

12. the non-utility of those questions requiring the validator 
- to assess non project-based data 

13. the inapplicability of certain questions 

14. the problem of responding to multi-faceted questions with 
single number weights 

15. the inequitable distribution of questions per section, i.e., 
1-2, 11-18, III-7, and IV-15, in light of the 25-25-25-25 
potential point distribution 

For example. Section II is v;eighted too heavily in favor 
of the project that has accurately processed inconsequential 
data. 

SMA is in understandable agreement with the identification of the 
areas of difficulty. The validators, however, have identified both 
problems within the Instrument, as well as difficulties with the 
philosophy of the procedure itself. We have culled out of the fifteen 
problem areas those eight areas dealing with the questions themselves, 
as well as three areas we believe a'^e tra^'ning problems, and the remaining 
number are problems of a philosophical or procedural nature. 

• problems with the Instrument per se 

a) the limitation of the Instrument 

b) the Instrument's dependency on v/ell stated objectives 

c) the need for clarification and redefinition of terms 

d) the need to revise the point values to reflect the 
effectiveness/exportability of the questions 

e) the resultant need (of changed point values) to revise 
the conversTon tables 

f) the inapplicability of several of the questions (as 
well as those questions requiring program evaluation 

O rather than the validation of documentation) 

ERIC 
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g) the need to ronove muHifPxeted questions and to 
replace them with singly focused questions 

h) the need to distribute more equitably the questions 
by section 

problems requiring emphasis in validator training 

a) assisting state coordinators and local personnel 
in the preparation of project documentation 

b) resolution of the issue of hov; behavioral objectives 
are to be stated (for purposes of the IVD procedure) 
and the areas the objectives must address, i.e., 
"cognitive" relative to instructional practices, and 

"process"relative to program management 

c) the form in which the validators are to receive the 
objectives and against which documentation is to be 
prepared, i.e., performance level statements, the 
relevance/meaningfulness of the performance levels 
(and of the objectives themselves) 

procedural/philosophical issues 

a) the IVD process presently assumed in the Handbook 
requires a research methodology with heavy emphasis 
upon some type of experimental design — pre/post testing 

b) There is no distinction in the Instrument between 
questions requiring the validator to make program 
evaluations and validation assessments. This confusion 

of roles biases the validator's response toward evaluating 
the program beyond what is presented in the documentation, 
e.g., the relevance of the stated objectives, assessing 
project information not germane to the objectives cited, 
and the conscious (or unconscious) desire to evaluate 
project management procedures even though not included 
in the documentation. 

c) the need to require project data on management procedures 
such that projects with primary contributions to manage- 
ment can be recognized, and/or projects with effective 
practices as a result of good management can be seen and 
validated in their entirety 

d) the question of how to correctly define the parameters 

of a successful project, i.e, "does the validator respond 
only to what can be documented or is he responsible for 
making a separate determination of the project's gestalt?" 
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It occurs to us that these four procedural /philosophical issues 
accentuate the dilemma of defining the validator's role. If the validator 
is to make program assessments (beyond written documentation submitted), 
then the process can never be reliable in the sense that all validators 
are responding to data in a uniform way. If, on the other hand, the 
validator is to respond solely on the basis of written documentation, 
or on testimony received from loca.l sources, then the projects v/ill 
suffer until such time that educators have become skilled documentarians 
and validators have thrown off their evaluation-oriented biases. Neither 
possibility appears likely in the near future. There is, nonetheless, 
considerable cause for optimism inasmuch as the IVO procedure is underway 
and there is detailed feedback and a bank of experienced personnel going 
into Year II development activities. 

Clearly the next step is the revision of the Instrument vis a vis 
these multiple criticisms. 

Additionally, as a developmental effort, there need be no demand 
that the Instrument "stand alone" as a validated document until educators 
cn all levels have had more experience in "identifying clearly the factors 
essential for success. In our opinion it is not desirable to insist that 
the Instrument be required to stand alor/e as if it were a nationiilly 
normed and validated procedure. The issues confronted in validating 
success on a cost effective replication basis are not unlike the practice 
of the law. The law does not stand without interpretation, and the entire 
legal procedure is tons tacitly in a state of development. The gap that 
needs to be filled bet\^/een the profession of a law and the profession of 
certified educational practices is that of developing quantification 
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procedures of general acceptability throughout the total educational 
community. The IVD Handbook is a first step in this direction. 



Part II 
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FINAL REPORT 
VALIDATION TRAINING FOR TITLE m, ESEA, PRACTICES 

I. INTRODUCTION 

" SMA/ESD personnel conducted twelve training sessions during the month of 
January 1973, as follows: 

Columbia University team for N.J. pilot test. Office of Education Title III and 

related Bureau personnel, 

and ten (10) regional training sessions for all the states except California and 
Vermont. 

These ten regional meetings were held in Silver Spring, Maryland; Seattle, 
Washington; Jackson, Mississippi; Oklahoma City, Oklahoma; Sioux Falls, South 
Dakota; Highwood, Illinois; Reno, Nevada; Atlanta, Georgia; and Windsor, Connecticut. 

SMA/ESD personnel were responsible for what was generally the second full day 
of a two day workshop. Our presentation covered the following areas: 

1. A theoretical introduction to the task of validation as presented in the "Hand- 
book for Validation of Educational Practices", December, 1972. 

2. The mechanical/logistical details of the validators being on-site, and con- 
ducting their documentation re\dew and assessment. 

3. A detailed exegesis of each question in the instrimient, as well as a thorough 
review of peirtinent material not covered in the Handbook, but responsive to 
needs arising from the N. J. field test of the Handbook procedures. 

Additionally, SMA/ESD staff prepared and distributed a "Guide for On- Site Vali- 
dation Team Procedures for Title m, E, S. E. A. Practices. " Over 550 copies of this 
19 page document were distributed. Included in the Guide was a Validator Self Analysis 
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Form (p. 12) which was to be completed by each validator and returned to U.S. O.E. 
in a return-addressed envelope provided by the contractor* 

As part of the training session, and included in the Guide , workshop participants 
were asked 43 logistical/mechanical questions about the entire validation effort. This 
instructional effort was undertaken using a flash card response enabling the SMA per- 
sonnel to immediately identify where clarification or additional instruction was needed. 
The questions in the Guide permitted each participant to make a permanent record of 
the correct response for use on-site at a later time. ^ 

A series of hand-prepared acetate transparencies were utilized on an overhead • 
projector to introduce workshop participants to the validation procedures and theory. 
These "overheads" were prepared "on-the-run" as a result of constructive feedback 
from the first two sessions in Silver Springs, Maryland. While the instructional con- 
tent remained constant throughout the ten regional workshops the method of presenta- 
tion was modified and expedited as a result of the preparation of the overheads, and 
the resolution of previously unanswered questions prompted by the participants. 

A following section provides a thorough analysis of workshop participant responses 
to the SMA/ESD presentation, and clearly indicates an affirmative reaction to content 
and presentation mode. These are gratifying findings particularly in light of the un- 
avoidable reliance on the spoken word as the primary means of communication. Addi^ 
tionally, there was unquestionably an "information overload" on the part of the presente 

^* Please note Guide attached. See pages 1-3. 
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It seems clear to us (SMA/ESD) that many of the concepts presented were new and 
somewhat demanding intellectually, and that upon post testing participants would 
have discovered less of a grasp of detail than anticipated. Preliminary findings from 
this first pilot year will clearly indicate those areas within the Instrument, and among 
the on-site procedures, which need additional clarification or definition. 

Finally, we were pleased with the general tone and conduct of the workshops. 
There was very little objection to the procedures proposed, and upon explanation of the 
"givenness" of the Instrimient, and the procedures, the pace and the atmosphere were 
cordial and accelerated over earlier sessions. While we were in no way responsible 
for the content or conduct of the first day of the two day workshop we received exten- 
sive feedback suggesting that the order of presentation be reversed. Iliere are argu- 
ments pro and con for this suggestion, but there can be no single best answer. Our 
recommendation for year II training workshops would be tliat: 

a) regional procedures be standardized 

b) project personnel whose practices are to be validated attend the workshops 

c) that team assignments and team leaders not be determined until after the 
technical (SMA) presentation 

d) that the "blue sheets*' be revised as a result of workshop attendance, and that 

e) the presentation for the workshops intermix the theory and mechanics of 
Instrument usage, with the logistical arrangements which states and local 
project personnel must make. Such a balance in presentation will make the 
first day's presentation more productive and will provide state coordinators 
with the detailed information they need to constmct their teams, and to ar- 
range for on-site procedures. 
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Finally, we received extensive feedback on each of ttie Instruments questions. We 
anticipate the opportunity of participating in a continuing contractual relationship with 
Plans and Supplementary Centers personnel in order to revise the instrument in light 
of these many excellent criticisms. Additionally, we at SMA/ESD have a number of 
structural recommendations to make relative to the field-use of the Instrument and 
Handbook, and for validation procedures for the Year II revision. 
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WORKSHOP PARTICIPANT RESPONSES 

Approximately 60% of those participating in the workshops responded to the Post 
Training Ses sion Reaction Form. This percentage represents 3S0 responses out of 
approximately 550 participants- This latter figure was determined on the basis of the 
Guides distributed. The Reaction Form was not utilized in the New Jersey Field Test 
nor for the training of Office of Education personnel. We further assume that the 
majority of those responding to the Reaction Form were actually those beir^ trained 
as validators for in only three workshops were large numbers of local project person- 
nel in attendance. 

The first question asked the particijjant to rank ten items on the following scale: 
4 excellent 
3 good 
2 fair 
1 poor 



The averages across all ten training 


sessions were 


as follows: 




Rank 


Percentage 


consultant's knowlec-ge of topic 


1 


3.48 


app rop riatene s s 


, 2 


3.25 


information presented 


3 


2.91 


handouts distributed 


4 


2.82 


materials presented 


5 


2.78 - 


length of presentation 


6 


2.76 


general evaluation 


7 


2-72 
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Rank 



Percentage 



method of presentation 



8 



2.58 



activities experienced 



9 



2.29 



quality of visuals 



10 



2. 19 



the "general evaluation^' of the approi)riateness and content of the workshop, 
items 1 through 5, average out to 3. 04. It seems reasonable to us to portray the 
figure in this light since items 6, and 8 through 10 represent methods of presentation 
which were constrained by the very short time line for preparation. As noted else- 
where in the Report many of the visuals were prepared by hand in response to 'ananti- 
cipated questions resulting from the initial workshop. 

A random comparison of regional responses indicates a high level of uniformity of 
response from workshop to workshop. This uniformity augers well for the standardized 
implementation of the validation process across the country, and as such fulfills a 
major objective of the training contract with SMA/ESD* 

Equally Important are the findings relative to the participant's self-analysis in 
terms of how well he understood the validation concept prior to workshop participation, 
and then the level of understanding as a result of participation. There is no way, im- 
fortuneately, to quantify these responses since they are highly subjective and immune 
to testing. Nonetheless the figures indicate that G6% of the participants categorized 
their knowledge prior to the workshop as "poor" to "fair", whereas 70% indicated 
their understanding after the work^rhop was "moderately" to "greatly" improved. In 
light of the professional training and experience the participants bring to the workshop 
process we find these figures highly interesting. One possible interpretation might be 



ERIC 



22. 

the newness of the content and the processea being undertaken. Another might be the 
distinctions made between validation in its statistical context juxtaposed with valida- 
tion as the process of verifying relevant documentation* In either event the number of 
participants expressing feelings of growth and the responses to "appropriateness" , 
"information presented", and "consultant's Knowledge of topic" indicate that the val- 
idation concepts were heard and responded to affirmatively. 
The actual figures are as follows: 

"Prior to the training session my knowledge of the Title lH validation process was 

Number Responding % of Total (N = 322) 
Poor 105 32. 6 

Fair 106 32.9 

Adequate 42 13. 0 

Good 58 18. 0 

Very Good 11 3. 5 



100% 

"As a result of attending this training session I believe my knowIe<%e, skills and 

abilities for conducting on-site validation procedures are:" 

Total number in % of 

Category Total (N = 32l) 

Not improved 11 3. 4 

Slightly improved 76 26.4 

Moderately improved 101 31* 3 

Greatly improved 125 38. 9 

100% 
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Readers will note the imusually high correlation between the pre-session "fair" 
and ''poor" categories and the post-session "moderately improved** and "greatly im- 
proved" categories. Similarly the pre-session 3. 5% specifying a "very good" know- 
ledge does not, by definition, allow for much improvement, and the "not improved" 
figure bear^:= out the correspondence. Those, however, identifying themselves as 
"good" in the pre-session analysis expressed some degree ( > 8.0) of improvements 
Please note attached charts. ' • 

A final comment on evaluation may be in order to facilitate the training process 
for Year n. We strongly recommend that consideration be given to designing the entire 
workshop program as an integrated experience. This means that personnel providing 
for the standardization of the training nationwide need to be built into the entire in- 
structional process, rather than addressing only the Handbook/Instrument on the 
second day. Many participants indicated that the first daj^ of training was ineffective 
without first receiving Instruction in Handbook procedures. 

We also recommend that the entire workshop be evaluated. By securing partici- 
pant responses to both days of training a better total instructional package can be 
developed. 
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Pre-Session Knowledge of Validation 



N = 322 



32.9 



u 
in 
z 
o 
a 
in 
u 
cr 

< 

O 

o 



25% 



18.0 



3.5 

V7777\ 




VERY 
GOOD 



32.6 



13.0 





75 %r- 



GOOD ADEQUATE FAIR 



Post-Session Knowledge of Validation 



N = 32! 



POOR 



50% 



3.4 

YZZZ2ZZL 



NOT 
IMPROVED 



31.3 



38.9 



26.4 






SLIGHTLY 
IMPROVED 



MODERATELY 
IMPROVED 



GREATLY 
IMPROVED 



CATEGORIES OF RESPONSE 
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m. LOGISTICAL PROBLEMS AND RECOMMENDATIONS 

Probably the major problem for the validation of practices this first year will be 
the inadequacy of comprehensive responses on the part of local project personnel on 
the blue sheets. These project responses will, we suspect, be Inadequate overall to 
the scrutiny required by our presentation. The validators, therefore, will undoubtedly 
be required to request substantial amounts of additional information. The transcrip- 
tion of this additional data (if it exists!), and the design of the data into the appropriate 
response format will probably be a cause for frustration, and local personnel responses 
such as "Why didn't they tell us all this in the first place!*' . Unfortuneately the "first 
place" we clearly perceived the problem was as a result of the N. J. test of the Instru- 
ment and the instructional procedures, and by that time (late December) it was too late 
to do anything other than recommend the findings for Year n development. 

The results of the Silver Springs meeting clearly identified the need for further 

clarification of the role of team leader. Additional data was built into the procedures 

during that first session to revise the role of team leader from arbitrator to co-equal 

member, and process facilitator. These additional responsibilities are noted on the 
2 

slide presentation . 

The results of the N.J. experience also indicated the need to identify, and to make 
part of the training process, the possible biases validators might bring to the docu- 
mentation/assessment process. These potential biases were identified and discussed 
in the training program. (Please note slide presentation for details. ) , 

2 

* See Appendix A 



26. 

We felt the data presented In the Guide was well received. We heartily recom- 
mend that this information be reviewed for Inclusion in the Handbook's Year II revision 
both in terms of the mechanics of the on-site visit, and for Instrument revision and 
improvement. Clearly the state coordinators need to be surveyed for their recom- 
mendations on ways to improve the logistical details since attention to this type of de- 
tail is as critical to effective validation as the validity of the Instrument itself! 

We recommend that a mini-Handbook be prepared for State* and local project per- 
sonnel to be sister documents for the materials for validators. Both documents clearly 
require numerous interfaces and internal consistency, but unless the local project 
people have better instructions and a longer lead time in gathering supportive docu- 
mentation remaking their case") the process will not work as efficiently as we believe 
it can. 

We also recommend that a Validator Bank be established with the names approved 
for inclusion chosen on a discriminating basis including, amor^ others, the following 
factors: 

1. The state coordinator's analysis of the validator's efforts on site, and the 
written results of that effort 

2. the analysis of the validator's written criteria (i. e. , the section for which 
he/she was responsible) 

3. Feedback from project personnel 

4. The assessment of the validator's self analysis on site, etc* 
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It became apparent as the workshops progressed that many of the validation pro- 
ceedings were being assimilated by some state coordinators for inclusion in their own 
application-for-funding procedures. We recommend that a panel be established to 
review those critical elements for validation which should be incorporated into state 
guidelines. Vs^lidation, for good or ill, rests squarely on a standardized information 
gathering and assessment/categorization base. It appears reasonable to us, therefore, 
that were the procedu: es to be standarized, and made imif arm throughout, the final re- 
sults will be more immediately useful. 

We also recommend that the Instrument revision process begin immediately for 
Year U utilization. We believe that modifications need to be made while the findings 
of Year I are clear and fresh in our minds, and that a greater amount of time needs to 
be allowed for field-testing the proposed Year II instrument. 

We further recommend that the "ownership" of the Instrument be concretized 
through the identification of representative personnel to sit on the revision panel, and 
that a minimum number of people, stipulated by name, be present from the larger 
panel for each successive revision session. 

As the validation procedure exists now the Instrument caimot stand alone. We 
believe that a strong argument can be made for not requiring that it stand alone! 
This position essentially argues (and modelled after jury proceedings) that the jurors 
and the "defendents" need to be instructed above and beyond what a written document 
can convey. Remembering that at least at present the end result of the entire pro- 
cess is a self-explanatory and free-standing document (Sections I- VI) the need for 
on-site assistance in interpreting the procedure will remsLin essential to its success. 
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At this point it is appropriate to note that the credentials and experience of those 
individuals chosen to serve as validators were of an unusually hi^ order* It is rea- 
sonable to assume, therefore, that if these gifted professionals cannot make a success 
of the effort this first time around, then in all liklihood no other group of professionals 
could either! 

Finally, we believe that a procedure needs to be developed and adopted, as a result 
of the Year II revision of the Instrument, (and the on-site procedures I) wherein the 
Division of Plans and Supplementary Centers personnel can have test data on how 
completely personnel selected to be trained as validators actually know and can dem- 
onstrate the required assessment skills! As with the developing procedures for the 
role of educational auditor there needs to be an agreed-upon competency base below 
which professional personnel not be invited to serve as validators. 

Competency data could be generated from several sources: 

1) testing (pre and post) 

2) assessment of personnel on site 

3) analysis of written validation reports 
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Part III Recommendations 

"Validation" represents fundamental theoretical and methodological 
departures from the ways in v/hich educators have traditionally evaluated 
programs. It Is our contention that validation deserves and requires a 
more comprehensive and internally consistent developmental plan if it is 
to he optimally responsive to the needs of the educational consumer. 
While giant strides have been made with this initial validation effort, it 
is clear from the feedback received nationally that mare intensive effort 
needs to be applied to the testing and resultant validation of the Instru- 
ment , and for comprehensive planning for Year II and following. 

Since the end product of the entire validation effort is the increased 
national assimilation of educational practices validated as exemplary and 
effective, it seems apparent that a strenuous effort needs to be directed 
to the complete and competent revision and implementation of the total 
process. Only exhaustive and coordinated efforts at Instrument and Handbook 
revision, and carefully planned and executed training workshops will result 
in the quality product required by the public for accountability, the 
Congress for continued funding, and the integrity of the process required 
for assimilation by the nation's educators. 

The following recommendations are made relative to IVD planning. 
Instrument revision, and training procedures based upon SMA's extensive 
involvement in the total validation process. These recommendations reflect 
our critique of the Validator Self Analysis Forms , the validators-prepared 
Reports, the team's participation in the Instrument's preparation, and 
the evaluation of the conduct of the training workshops. 

Readers are respectfully directed to Section I of this Report for 
^ detailed recommendations on Instrument revision, as v;ell as Section III-B, 
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and to the Appendices for a proposed time schedule for the development of 
Year II validation procedures. 



A. Validation Planning and Training Procedures 

1. We strongly recommend that a representative Panel be selected 
to work with the training and Instrument revision contractor; 
that the Revision Panel contain twelve members stipulated by 
name, and that the Panel not meet to take action on the 
validation process unless a quorum is in attendance; and, 
that, further, the Panel "own" the results of the revised 
procedure including both the Year II Instrument, and the 
operational procedures themselves. 

2. That the Review Panel's advisory duties would include, among 
others, the following tasks: 

1) arbiters of a Section's point value, and of the 
value of each question within a Section 

2) assisting in identifying appropriate sites for 
field testing the revised Instrument 

3) as an advisory group to respond to the contractor's 
recommendations, and/or their ov/n, for the inclusion 
of new questions/sections 

4) the approval of the revised Handbook , training 
procedures, and the Instrument 

3. That the BESE and/or Plans & Supplementary Centers personnel 

hire one firm as prime contractor for Year II revision activities, 
and that said firm be charged with the responsibility of planning 
and implementing Year II activities including the revision of 
the Validation Instrument 

4. The presentations at t'';e ten {+-) workshops be conducted by one 
contractor for the sake of uniformity, and, that the training or 
"content" portions of the workshop be interfaced with state/ 
validator organizational concerns. This proposed balance in 
presentations will allow discussion immediately of both the 
on-site logistical details, and the demands theoretically and 
Instrument-wise of the validation undertaking 

5. That a simulated learning package be prepared for use in 
establishing coder reliability for practice and testing in the 
training workshop, and, further, that as a critical aspect of 
the field-testing of the Year II IVD Instrument that "back-up** 
teams review the findings of the earlier team, and that both 
sets of results and scores be matched for the identification of 

Q coder discrepancies. Such follow-up t^am validation assessments 




v/ould need to be conducted on a shorte. term and randomized basis. 
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The entire workshop should be evaluated, and not just the 
performance of the training contractor- We further recommend 
that the state coordinators elicit feedback on the entire 
selection and on-site visitation procedure. He also strongly 
recommend some form of pre/post testing (of those selected to 
be validators) for feedback on v;hat their understanding of 
the validation task may be, and for the identification of 
problem areas to be addressed in the workshops as a result of 
the pretesting. 

That state coordinators be encouraged to conduct a post mortem 
session with both validators and project personnel for feedback, 
and that such capta be related to IVD personnel in Washington 

Project personnel v/ith practices to be validated' should be 
invited to attend the Validator Training Workshops 

That the revision of the Handbook be directed to: 

1) the preparation of a complete glossary of all IVD 
terms in the Instrument 

?.) the inclusion of a section on the theory of validation 
as the proposed rules for evidence reviev;, and the 
critical distinctions the IVD process makes between 
evaluation and validation 

3) revisions in the procedures for team member selection 
and assignment 

4) more complete and sensitive directions for local project 
personnel in the completion of the blue sheets 

5) the functions of the team leader, and complete details 
on the recommended team interaction process 

6) the advantages and disadvantages of team observation of 
practices 

7) a discussion of possible validator biases 

8) a comparison of the roles of IVD validator and educational 
auditor 

9) detailed instructions on preparing the Validation Report 

10) the procedures for conducting the on-site visit including 
a proposed time schedule 

11) the specification of the state coordinator's responsibilities 

12) the specification of the regional coordinator's responsibilities 
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10. That pertinent technologies and methodologies to increase 
dissemination/exportability potential be included, and with 
specific reference to: 

1) instrumentation addressed to decision-settings, 
evaluation formula, and planning models 

2) cost formulations on a per pupil per instructional 
- hour basis 

3) cost conversion scales for the geographic comparison 
of costs for potential consumer districts 

4) instrumentation addressed to identifying the qualities 
required for effective leadership in project 
replicability, i.e., how is the charismatic leader's 
behavior to be analyzed in terms of actions essential 
to success in the consumer district 

5) a system to code "practices" against consumer needs, 
i.e., by academic area, size, staff/student ratios, 
costs per pupil, futures orientation, demographic 
descriptions, etc., all directed to providing 
strategies for educational change 

11. That adequate time be allowed for both the field testing of 
the revised Instrument, and for the training of local project 
personnel in the preparation of documentation (for use on the 
blue sheets) 

12. That a "Mini -Handbook" be prepared for state and local project 
personnel citing case studies and other illustrative data of 
v/hat constitutes acceptable documentation, the proper form for 
objectives, appropriate examples of evaluation designs, 
management instruments, and testing procedures, etc. Such 

a Mini-Handbook would greatly facilitate the completion of the 
"blue sheets", and would facilitate the on-site review of 
documentation 

13. Validator team assignments should not be determined until after 
the training v/orkshop is completed — particularly with respect 
to the selection of the chairperson 

14. That a Validator Bank be established in order that trained and 
experienced personnel will be on record for use by the states 
as the IVD process grows and, further, that a procedure be 
developed for certifying said validators employing conipetency 
data from at least the following sources: 

a) pre and post testing 

b) assessment by state personnel of their work on site 

c) analysis of their written validation reports 
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15. Procedures for identifying potential validators should be 
uniform throughout the ten regions 

16. That the SMA proposed time schedule be adopted with such 
modification as may be necessary 

17. That instructions for formating validated project findings 
for submission to ERIC be included in the revised Handbook 



Instrument Revision 

1. That the Instrument be thoroughly revised as. a result of 
BESE, SMA, validators and NASACC criticisms 

2. That the blue sheets be correspondingly revised 

3. .That the Instrument be revised to show "profile" data that can 

be optically scanned and computer tabulated for rapid classifica- 
tion 

4. That the Instrument request data indicating both the need, and 
the state-wide priority ranking of the need to which the project 
is an effective response 

5. That the detailed criticisms synopsized by the SMA analysis 
of the Validator Self Analysis Form be addressed in detail 
Please see Section I of the Report. 
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Introduction 

"Validation" represents fundamental theoretical and methodological 
departures from the ways in which educators have traditionally evaluated 
programs* It is our cortention that validation deserves and requires a 
more comprehensive and internally consistent developmental plan if it 
is to'be'optimally response to the needs of the educational consumer. 
While giant strides have been made with this initial validation effort 
it is clear from the feedback received nationally that more intensive 
effort needs to be applied to the testing and resultant validation of 
the Instrument , and for comprehensive planning for Year II and following. 

Since the end product of the entire validation effort is the increased 
national assimilation of educational practices validated as exemplary and 
effective it seems apparent that a strenuous effort needs to be directed 
to the complete and competent revision and implementation of the total 
process. Only exhaustive and coordinated efforts at Instrument and 
Handbook revision, and carefully planned and executed training v;orkshops 
will result in the quality product requirti by the public for accountability 
the Congress for continued funding, and the integrity of the process 
required for assimilation by the nation's educators. 

SMA is in a uniquely advantageous position to undertake the proposed 
tasks outlined in this Prospectus. First, SMA/ESD personnel have been 
deeply involved in the generation of the first Instrumen': . Second, SMA/ESD 
personnel prepared and conducted the national workshops for Instrument 
utilization. An essential aspect of the conduct of the workshops v/as the 
collection and analysis of dotailed responses to both the mechanical pro- 
cedures in the validation on-site process, and to needed changes in the 
Instrument . Critically, SMA/ESD personnel are the only individuals v/ho 
have been involved in the entire process from procedures development to 
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final evaluation for Year !• The participant critique of SfWESD 
personnel in the Workshops indicates a high confidence level for 
continued participation in the developmental effort. Additionally, the . 
corporation's professional assets and capabilities make SMA a logical 
choice as prime contractor for the expanded development and improvement 
of the national validation effort J 

Cognizant of the need to maximize producer/consumer interaction 
through the vehicle of educational validation SMA proposes to deliver 
the following products and services. The delivery schedule proposal 
in Section VII will need to be negotiated, and dates determined. 

The following sections reflect SMA's professional judgements on the 
need for Handbook and Instrument revision and for needed improvements 
in implementation. These six sets of items represent our responses to 
extensive feedback from local, state, regional and federal personnel 
during the conduct of fourteen (14) training sessions coast to coast* 
The responses are also reflective of those needed logistical details that 
v/ould only be apparent to a contractor faced with training personnel in 
the use of the Handbook . 

We submit these items for consideration fully aware of the cost and the 
thousands of professional man-days of effort committed to the process to 
date. Our criticisms of existing processes are for the sole purpose of 
improving the validation process and for expediting the assimilation of 
cost-effective educational practices. 



1. Please note SMA/ESD capabilities in Section IX. 
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Handbook Revisions Scope of Services and Products for Delivery 



A) Supervising Panel for Handbook Revisions 

We propose the construction of a representative panel of 
t^'elve people to oversee the revision of the Handbook , 
The panel, v/orking in close relationship with SMA, would 
serve as: 

1) arbiter of a section's and a question's relative value 
using an appropriate weighting procedure (Q Sort, Delphic 
survey, etc.) 

2) an arm in selecting appropriate sites for field tests 
of the phase II revisions 

3) assistance in identifying teams to conduct document analysis 
as a validation of previous team findings 

4) the approval body for the successive revisions of the 
Handbook and Instruments 

B) Logistical/Mechanical Revisions 

SMA personnel will revise the Har>dbook's procedures to reflect 
the actual implementation of Year I validation proceedings, 
as well as to include suggestions for improving implementation 
for Year II. It is apparent to us that the improvement of 
the mechanical details of the total validation effort is just 
as critical as revised and improved instruments. In the 
rev;riting of the Handbook's white pages careful attention will 
be directed to: 

1) the preparation of a complete Glossary 
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2) the inclusion of a section on theory of validation 
as the rules for evidential review, and the critical 
distinctions between validation and evaluation 

3) revisions in the procedures for team member selection 
and assignment 

4) more complete and sensitive directions to local project 

V 

personnel in the completion of the Self Analysis Form 
(the blue sheets) 

5) specific and scheduled training workshops for those 
serving as validators 

6) the functions of the team leader, and a complete 
itemization of the team interaction process 

7) the advantages and disadvantages of team observation of 
practices 

8) potential validator biases 

9) the role of validator compared with the role of educational 
auditor 

10) instructions for preparing the validation report 

11) the procedures for on-site validation including a time table 

12) a more coherent outline for the final validation Report, 
including instructions to local and state personnel on required 
content and format 

13) the responsibilities of the state coordinators 

14) the responsibilities of the regional coordinators, and, 

15) detailed attention to the content and scheduling of the 
^ training workshops 
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C) Instrument Revision 

SMA personnel will present to the Panel a revised Validation Form 
for on-site use. This revision will reflect the multiple changes 
needed to give the Instrument greater strength, academic 
' credibility, and, after field testing, increased validity and 
reliability^ This revision, once approved by the Panel, v/ill 
be employed by multiple teams using the same documentation to 
check for coder reliability. Revisions resulting from this 
internal validation will then be proposed for inclusion as the 
revised instrument for Phase II. Please note the proposed time 
schedule for implementation in Section VII. 

D) Project Nomination Form 

As a concurrent activity SMA personnel will revise the blue 
sheets to correct identified weaknesses in Phase I, and to be 
compatible with the revision of the On-Site Validation Form. 
Revisions will include formating, clarification of the questions, 
and examples of desired materials. 

Preparation of Content for and Operation of Regional Training Workshops 
A) SMA will schedule workshops for state and project personnel, those 
to serve as validators, and BESE personnel. 

Content for these workshops will reflect all revisions and new 
procedures. Workshops for all levels of personnel can be held 
in the same tim.e period. This overlapping of levels of personnel 
will result in improved documentation and accelerated on-site 
visits. Additionally, this multiple targeting approach to 
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conducting the workshops will accelerate the dissemination 
of effective practices regionally, and the inclusion of 
validation procedures in state application forms for funding 
new projects. 

B) SMA will prepare in narrative for, for inclusion in the Handbook , 
the desired format for the conduct of the workshops interfaced 
with the responsibilities of federal, regional and state . 
coordinators. 

III. Special Work Tasks 

SMA personnel will propose for the Panel's consideration pertinent 
technologies and methodologies to enlarge the dissemination and 
importability potential of effective educational practices. These 
additions would include: > 

a) instrumentation addressed to decision-settings, evaluation 
formula, and planning models; 

b) cost formulations on a per pupil per instructional hour basis; 

c) cost conversion scales for geographic comparison; 

d) a package of simulated training experiences for validators to 
be used as a screening device in identifying potentially low- 
effective validators; 

e) the procedures to establish a Validator Bank of certified 
professional personnel including criteria for selection; 

f) instrumentation addressed to identifying the qualities required 
for effective leadership in project replicabili ty; 

g) a system to code "practices'* against consumer "needs", and, 
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h) a detailed cost analysis comparison of how to train the 
maximum number of personnel on all levels at minimum cost 

IV. Assessment and Classification of Year I Validated Projects (Practices) 

SMA will: 

a) identify response problems 

b) classify reports by category and type 

c) coordinate findings for computer access with an appropriate 
computer installation (e.g., Kentucky Title III Project) 

d) make recommendations on dissemination format, and 

e) preipare ERIC Abstracts 

V. The Preparation of Validation Guidelines for State Application 
Procedures 

SMA proposes that key elements of the documentation process be 
modified for inclusion in State application procedures. SMA will 
prepare such guidelines for adoption by interested states. The 
adoption of these guidelines will facilitate the validation of 
educational practices and will expedite the matching of effective 
practices to particular learning needs. 

VI. Publications Production 

SMA's publishing subsidiary Scientific Management Publishers, is 
capable of producing all printed matter required, at competitive 
prices, and on a very short time schedule. Possible options for 
publication: 
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a) the revised Handbook 

b) supplementary materials for the conduct of the workshop 

c) materials for the Educational Fair 

d) materials to mail to school districts requesting 
additional project data 
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10. 



VIII. Time Line anu Costs 

SMA is proposing the adoption of the services in Sections I 
through V as the basic Phase II contract package. It occurs 
to us that these are the essential components of the total 
validation process, and that all of these elements need to be 
addressed as a synergistic who'^a. Section VI provides for 
publication services. Costs would be determined by the size 
and format of the material in question. 

Section VII proposes a sequenced list of delivery dates 
responsive to the problems encountered in Year I, and provides 
for adequate lead time in completing each task and meeting each 
critical interfaced 

Costs for services are determined by computing man days of 
effort, overhead, general and administrative expenses, material 
and supplies, and profit. Specific costs v/ill be prepared for 
submission along with a complete proposal responsive to those 
items negotiated for inclusion. 

The formal proposal will include a PERT network, including 
subsystems for materials classifications, data processing for 
computer access, and printing. 



1, "Interface" - a term from PERTing indicating an essential decision 
point, or conjunction of critical activities. 
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Content for Validator Training Workshops 
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SELF ANALYSIS INVENTORY ON THE HANDBOOK 
FOR VALIDATION OF EDUCATIONAL PRACTICES 







True 


False 


1. 


It is essential that the validator be able to identify and describe in 
his own words the specific educational practice under examination. 






2. 


The validator should have read the entire Handbook prior to the 
on— site vi'?itation. 






3. 


It is the primary responsibility of the validating team to judge the 
cxtc^iiLy relevance anu auLuenticiiy oi tne uata suDmitteu oy ztiQ pro- 
ject as evidence of the successful operation of the practice under 
scrutiny. 






4. 


There is a minimal number of points that a practice must receive 
for each criterion in order for the practice to be nominated for 
adoption. 






5. 


It is essential that each validator rating include the identification 
of evidence and rationale for each number-weighting. 






6. 


Validators are to see local project Self Analysis form ratings. 






7. 


Validators are to see local project and S.E.A. question responses 
and citations of evidence. 






8. 


^'Validation*', as the term is defined in the Handbook, is used in 
a non-statistical manner and rather means . . reviewing a 
practice to verify its credibility as an exemplary^ program,'' 






9. 


The minimal number of validators for each project's practice 
(or practices), z-egai^dlcss of size, is three. 






10. 


Validators arc cho'jen by the Title III regional coordinators for 
each state project. 






11. 


"Exportability *' is the new "in" word for high dissemination 
potential. 






12. 


Validators a'»"3 selected for teams on the basis of experience 
germane to the section of the instrument for which they have 
primary responsibility. 






13. 


"Validation", as the term is employed in the Handbook guarantees 
the replicabilit}^ of the educational practice in other settings under 
similar circumstances. 






14. 


It is the team chairman's responsibility to prepare the narrative 
report on each of the four criteria in the instrument. 
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True 


False j 


1 K 
JLD. 


ii IS tne teaiii iiiejTiJjei s responsiDiiii^ to proviac sucn a.uuiLLona.i 
data as he feels is critical to the validation procedure even though 
such data may not have been called for in the questions. 






IG. 


The entire team must be in complete agreement on each of the 
four criteria for the practice to be recommended for national 
adoption. 




1 


17. 


One may serve on a validation team without having attended a train- 
ing session. 




j 
I 


IS. 


Validators unable to attend their own scheduled training session may 
attend a session scheduled for a neighboring region. 




1 

1 


19. 


The primary responsibility of the validating team is the review of 
the practice's data. 




1 


20. 


The validation procedure requires that each team member observe 
each of the practices imder scrutiny. 




— 1 


21. 


The validation team must review its findings and conclusions with 
project personnel prior to final departure from the site. 






22. 


Team members may request such additional information as they 
may need in responding to the questions « 




1 

1 




i lie ueani cnaiirnan rnust ouuniiL d. rougn urd.iL ui Liit; iinai r^jpuiL 
to the other members of the team prior to final departure from 
the site. 




— 1 

1 


24. 


The final report includes a narrative summary of each of the four 
criteria, plus the validator's comments and evidence on each 
criterion. 






25. 


Sections VI and VII of the final Validation Report are the sole re- 
sponsibility of the validation team. 






26. 


The completed Validation Report is tO be submitted to U.S. O..E. 
within 10 days of the on-site visit. 






27. 


The state coordinators complete the Checklist in Chapter V-C 
and submit same to U.S.O.E. 






2S. 


Data other than that presented by the project for validation may be 
included in the Validation Report. 






29. 


The primary criteria for an innovative practice is that it be exer- 
cised in less than 10% of the school districts in the regie:, in question. 






30. 


The ^'realism'' of a particular objectiv^e's performance level is r.imply 
some increment of gain over tra.d;tional practices. 
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31. The "app* priateness" of a given evaluation procedure is the 
congruency between the stated objectives, their attendant activ- 
ies, and the characteristics of the learner. 

32. Instrument "validity for the purposes of measuring a given 
objective, Is essentially a question of content, construct, and 
face utility. 

33. Instrument; "reliability", for the purposes of measuring a 
given objective, implies a high correspondence between the 
instrument's accuracy and its stabili^iy under repeated usage. 

34. A "causal comparative analytical technique" is one which fits 
under no other more precise heading. 

35. The modified Likert scales utilized in this Handbook require the 
validator to set up his own constraints - for weighting his 
responses - solely on the basis of the evidence in hand. 

36. Questions demanding judgements of "extent", "accuracy" and 
"appropriateness" are to be made solely on the basis of the in- 
formation provided by tho project. 

37. The team chairman is to include a one page narrative statement 
on those areas of the practice considered critical to the prac- 
tice's success, but not covered by the instiaiment. 

38. Difficulties encoimtercd in utilizing this pilot validation effort 
(the Handbook) are to be fully noted and sent to the U.S. Office 
ot Education's Division of Plans and Supplementary Centers. 

39. Prior to observing the actual practice under nomination the 
Validation team should review the amoimt of data submitted 
for scrutiny. 

40. The validation team should have a schedule for the day identi- 
fying tho time to be committed to obser\^ation, data analysis, 
interviews, team consultation, exit interviews, and review of 
rough draft of final report. 

41. Each team member is responsible for preparing a narrative 
summary of his particular section of tho Report. 

42. States wishing to include their exemplar^ practioos in the 

Ed Fair must participate in chc U.S.O.E. - directed validation 
procedure. 

43. The U.S. Office of Education's Division of Plans and Supplemen- 
tary Centers is the principle source of the items in the Handbook. 
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ON-SITE VALroATOR CHECKLIST 



Note: It is recommended that this Checklist be utilized on the evening prior to the all-day 
validation effort. Any negative responses should be addressed before the validation 
activity gets underway. 









Yes 


No 


1. 


Have you thoroughly read the entire Valid^^ition Handbook? 






2. 


Have you read, and do you have in your ])ossession, the following 
documents ? 








a) 


original proposal(s), addenda, and continuation agreements 








b) 


the completed Self Evaluation and Project Nomination Foi >n 








c) 


the complete project evaluation plan 








d) 


evaluation data from testing 








e) 


data analysis procedures utilized 








f) 


names and qualifications of personnel involved in test admin- 
istration and analysis 








g) 


educational/instructional materials produced by the project 






3. 


Are you familiar with the Staters applicaticn and project evaluation 
procedures ? 






4. 


Do you understand your rol6 on the team / 






5. 


Are you familiar with the State's on-site financial and audit reports 
on the project? 








Are you familiar wdth the State Advisor^' L-oard^s reports (where 
applicable) on the project? 
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PROPOSED DAILY SCHEDULE 

Evening of arrival 

PM Team introduction (The team, state and local project personnel) 

secure and review team ir^ember folders 

Review team member and total team responsibilities 

Complete On-Site Validator Checklist (and address any areas of 
need identified) 

Day on-site validation 

AM 8:00 Meet all projecu personnel, and secure work space 

8:10 Analyze quantity of data to be reviewed (printed materials, visuals, etc*) 

8:20 On the basis of the data to be analyzed schedule the remaining work day 
(observations, interviews and review of written materials) 

8:30 Obser\''ation of the educational practice (s) if appropiate) 

9:15 Review of project data utilizing The Handbook's On-Site Validation 7?orm 
(yellow pages) 

Lunch 

PM 1:00 Continued work on On-Site Validation Form : 

3:00/3:30 Team meets to reacli agreement on each section (criteria) 

3:30/4:00 Each team member preparvcs a rough draft narrative summary of his 
section 

4:00/4:30 Team meets with project/state personnel for exit interviews 

Team cliairman submits rough draft of final report to teum members for 
sign-off 

4:45 Team members fill out V£*iidator Self Analy-sis Form and pxaca in mail. 
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L Team Procedures 
Introduction: 

The special task confronting the validation team is tha t of assessinjj the utility- of the 
infuvniation presented in terms of its credibility as evidence in validating an exemplaiy prao- 
tioo^ ■ The burden of the task, therefore, is to wcdgh the multiple evidence presented in terms 
of tlie judgments required by each of the questions. Clearly this is not the traditional approach 
lo evaluation generally practiced in the nation's schools. Rather, the purpose of the validation 
cfiort is to verify the credibility of the project's practices and the reports on those practices. 

The very fact that a practice has been nominated constitutes its ihnovativeness and suc- 
ce.ss on the state leveU It is, therefore, the team's responribility to determine if what the 
project personnel said was happening, was in fact happening, on the basis of the tangible evi- 
dence submitted for review. 

This evidential approach is critical to the practice's adoption since it is on the basis of 
the team's review and summaiy of the written documentation that othei' districts will have ac- 
cess to the information. 

It is clearly necessaiy, therefore, that the team review the evidence both individuallj' 
find collectively. Tho team chairman has been instructed to allow time for total team con- 
sultation prior to tlie preparation of the Validation Report in order that tliere may be a good 
interchange of data on the total operation of the practice within its institutional setting. 
IL Instrumentation 

The On-site Validation Report Form hps been purposely designed as a self-contained unit. 
No additional interpretive nor recording data is needed or desired. Each question is self- 
contained, requires its own data ha£e, and is separately rated. Clearly the individual vali- 
dator will be responsible for setting his own parameters for rating judgments based upon Uie 
availv-bilify of material. Because this is so, and the subjective judgment of the validator is 
tiiu ciily basis for making response, it is necessai^^ to document the basis on which the decision 
wa.s made, and to cite critical evidence. It is clearly also evident that the validator muse make 
the judgment as to the r-idequacy of the data presented for review. 

As indicated in the Guide for On-Sitc Validation Procedures it is obviously essential that 
t^ach validator be able to describe in his own words the major objectives for the practices being 
validated. Tho major objectives are the sole basis of the questions, and all judgments must be 



nmde with the iraiidauon of the objective as the single concern. Where a validator feels 
strongly that notation should be made of particular processes utilized in the success of the 
practice such information should be so noted in the narrative coxnment. 
Recommendations for weighting responses: 

1. For each question we propose four (4) headings for reflectiori by each individual validator, 
and then for the team as a whole: 

a) within the parameters of the question are the exhibits adequate to cover the 
relevant variables ? 

(Please Note: as a framework of reference variables may be classified under three headings: 
1) behavioral; 2) instructional, and 3) institutional. The behavioral categories include the 
three domains - cognitive, affective and psychomotor; the instructional variables include or- 
ganization, content, method, facilities and costs; and the institutional variables include pro- 
files on students, teachers, administrators, educational specialists, families and the com- 
munity. ) 

2. Does the documentation (exhibits, evidence, etc.) support the nomination of the practice 
for national visibility? Is the documentation inclusive enough tj rllow for adoption? 

3. Is there high coder reliability? Has each validator requested and received feedback 
from his team mates on their weighing of the questions in his section? Is there reliability 
of response ? 

4. Does the team's su' inarization of the practice in question provide all the critical data 
needed for the adoption of the practice? 



The team chairman might find it useful to adopt a procedure such as the matrix below for 
weighing the team's response to each question, and for making judgments as to the adequacy 
of the supportive documentation and exhibits. 



Validation Check Sheet 



Section # 
Question # 



a paraphrase of the question: 



3. 
4. 



Indicators 



materials cover relevant 
variables ? 

documentation adequate ? 

high coder reliability? 

Validation Report inclusive of 
all critical data? 



Other considerations 

In order to regulate to the maximum degree possible the conduct of the on-site validation 
procedures the following suggestions for data analysis are being put in each validator's hands. 
As earlier noted, however, the "standards*^ within which Ihe validator determines a number 
weighting for the question under scrutiny is totally dependent upon the scope and extent of the 
materials available on the day of the on-site visit, 
I. INNOVATIVENESS 

Innovative practices may be one or a combination cf the following items: use made of a 
particular product; pariicular instructional procedures; pj>rticular organizational approaches; 
vmusual staff configurations; unique instmctional climate; and imusual applications of traditional 
materials or procedures. As noted on each criteria title sheet a definition of tlie key term 
imder investigation is provided. This clarification of the particidar terra is the interpretation 
to be used in answering the questions in this section. Other pertinent questions are also in- 
cluded in the boxed-in headings* . 
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II. EFFECTIVENESS 

The critical concern for this section is the extent to which the practice's objectives have 
been achieved and/or the learner's performances improved. Validators will want to consider: 

a) extent objectives are critical to understanding success of the practice; 

b) extent performance levels are challenging and realistic; 

c) congruency between objectives and related activities; 

d) appropriateness of test selection; 

e) reliability of test ^idmioistration; 

f) range and variety nf instn irr^jnts employed; 

g) appi'opriate data treatment procedures - descriptive,, analytical, inferential, 
comparative; 

h) relevance and imagination of evaluation design; 

III. COST EFFECTIVENESS 

This section posits as the standards against which effectiveness judgments are to be made 
th^- parameters of the data provided by project personnel. In sh'^vt, these judgments can only 
be made in response to the performance levels achieved compared with the costs per pupil. It 
is possible th?A the validator might be assisted in this task by requesting that project personnel 
prepare cost figures on a per pupil per instmctional hour basis. If this is not possible then 
the yalidator must estiniate the peiformance levels achieved against the expended ousts. 

The vr.lldator will also want to carefully review the costs presented, and to make some 
ji^dgments as to the completeness of the data. Where irregularities occur the validator may 
request primary sources. 
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IV. EXPORTABILITY 

The following considerations might prove helpful in responding to this secKom 
Will the practice be continued? Is the evidence for continuation encouraging 

2. Is there a high relationship between the local school district's use of the practices 
and the needs of the state at large? 

3. If applicable is there evidence of support by key constituencies ? 

4. Is planning, management and dissemination information adequate, clear and 
replicable ? 

5. Are critical processes and procedures well docu^'iiented and critiqued? 

6. How adequate was the identification of problems and the procedures for their 
resolution? 

7. Will the data submitted by projec \ staff, supportive and/or critical S.E.A« documen- 
tation,, and the Validation Report of the team, along with attachments, serve the 
critical information needs of adopting districts ? 
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Team Interaction and The Preparation 



of the Validation Report 



The team is responsible for completing all the questions in Section VII of the Handbook, 
It must be emphasized that each question requires both an explanation for the number weight 
assigned, and the citation of evidence reviewed in making the decision. The total team's 
responses will be included in the Final Report to be sent to Washington* The name of the 
validator responsible for each section of the Report is to be clearly identified. 

Section VI of the Report is the narrative summary for each of the four sections on Innovative- 
ness, Success, Cos<,-effectiveness and Expo r lability « This narrative' description of the prac- 
tice's objectives, opei*ation and evaluation must be comprehensive enough to provide an in- 
terested school district with all the information needed for adoption. (The U.S, Office of Educa 
tion does not at this time plan to disseminate Section Vn to adopting districts. This informa- 
tion will be inade available, however, on request.) 

In the process of preparing both Section VI and Vn the team will need to: 

a) complete each individual section and write a narrative summary of findings. 

b) meet as a total team to weigh each of the other three sections of the Handbook. 

c) review as a total team coder reliability for each section, and then make res; onse- 
weight comparisons with the team member assigned to that particular section, 

d) discuss areas of coder discrepancy. 

e) seek, where desirable, additional data on the issue under contention. 

f ) agrei2 to a point total for each section in the order prescribed in the Handbook. 

g) team members disagreeing with the majority opinion maj' prepare a dissenting 
report for inclusion in Section VL 

h) the team chairman collects the individual narrative summaiies on each section, pre- 
pares such introductory data as required, and draftis the final written report. 

i) the rough summary is shared with team members and each team member signs-off 
on the rought draft. 

j ) the rough draft is shared with project personnel as the essential content of the exit 
inten'iew. 

k) the cliairman takes the team-approved rough draft and prepares a final typewritten 
report. 

1) the chairman mails a copy of the Final Report, along with all four Section VH respons 
to the State Coordinal:or within 10 days of the visitation. 

m) cacli team member fills out the Validator Self Analysis Form at the conclusion of the 
^ on-site visit and mails the form to U.S.O.E, 
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Validator Self Analysis Form 



Please fill in the foUowring information at the conclusion of the on-site validation and mail in 
the stamped envelope attached. 

Name . Date 

Project Reviewed for Validation— . . 

City State 

Your Address 

City State \ 

Section Reviewed 

1. Do you feel your involvement reflected your area of expertise? 
Yes iio 

i 

2. Were the point values for the section you validated generally acceptable to your teammatti ' 
Yes No 

3. Did you find the task of assessing the data and weighing the responses difficult? 
Yes No 

Comments: 



Were there questions in your section 
terms of assigning number weights ? 
If "Yes'^fl please indicate section and 



that you foimd particularly difficult to answer in 

Yes No 

question numbers: 



5. Do you feel that there was adequate team interaction and discussion in reaching a conclu- 
sion on each of the four sections of the Report? Yes No 

G, Please comment on areas of difficulty with respect to both the validation instzrument, and 
the team's interaction with one another and with project personnel* 
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Exeirplary Practices and Decision Settings 



Note: The use of the following materials is voluntary. Where validators wish to categorize 
the impact of the pi7actice(s) reviewed for inclusion in the written report the foU.owing headings 
may be foimd useful.^ 

2 

These headings are based heavily on the work of Braybrooke and Lindblom , and provide 
a conceptualization of four generally different decision settings in education. Since the educa- 
tional practices under scrutiny have been identified as successful and exemplaiy by the State 
aulhorities th-^re may well be a h ^h correlation between the success of the given practice and 
the decision-settings in which the practice occurred. 

A decision setting . .is the total set of environmental circumstances governing both 
analysis and choice. "3 These four headings are differentiated through the intersection of two 
continua: small versus large educational change, and high versus low information gra-^ p to 
support change. 

The decision-making headings are as follows: 

1. Homeostasis: a decision-making setting characterized by decisions to 

effect a sniall developmental change supported by a high 
level of relevant information grasp. 

2. Incrementalism: a decision-making setting cliaract ;rized by decisions to 

effect a small developmental change supported by an 
initial low level of relevant information grasp. 

3. Metamorphism: a theoretical decision-making setting characterized by 

change based upon decisions to effect complete cliange 
in a system supported by high information grasp. 

4. Neomobilism: a decision-making setting characterized by plans to 

effect large change supported by an initial low level 
of relevant information grasp. 



Educational Evaluation and Decision Making, The PDK National Study Committee 
on Evaluation, F.E. Peacock Publishers, Inc., 1971, pp Gl-79 

2. 

Braybrooke, D. , and Lindblom, C.E. ''A Strategy of Decision," The Free Press, 
New York, 19G3 

Op. Cit. , p 61 
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Decision-rnalang Settings 



Homeostasis 


Metamorphisra 


PurpOkSe: maintenance 


Purpose: 


complete change 


Activit/: restorative 


Activity: 


Utopiai;! 


Basis: tecluiical standards 


Basis: 


overarching theory 




Model: 




Model: 


Evaluation formula: 


Evaluation formxila: 






Incremental 




Neomobilism 


Purpose: continuous improve- 


Purt>ose: 


inventing, testing; awd 


ment 




diffusing solutions to signif- 


Activity: developmental 




icant populations 


Basis: expert judgment plus 


Activity: 


innovative 


structured inquirj' 


Basis: 


conceptualization, heuristic 


Model: 




investigation, and structured 
inquiry 


Evaluation formula: 


Model: 






Evaluation formula: 



o 



Small 



Degree of change 



Large 



These settings, along with their appropriate models, allow the change-classification of 
educational practices on the basis of how the educational public perceives two Important 
variables: 1) does the commuriity (administrators, teachers and parents) view the variables 
to be altered as important?, and 2) does the commxmity view the magnitude of the change 
(proposed or completed) as trivial or important? The classification of a practice's impact 
for change, therefore, is not based on the magnitude of the change per se, but rather upon 
the commimify's perception about the change,. 



Finally, each decision-making setting ha? a correspondent model. The setting largely 
determines the decision-making model to be employed - whether consciously or unconsciously , 
Tliree models have been identified by Brs.ybrooke and Lindblom.^ 



Op. cit. , p. 48 



1- Synoptic ideal: a oecision model appropriate in .jdified form for homeostatic 
decision settings cllaracterized hy a high degree of comprehensiveness* 
It is termed ideal because it Is almost ne:^er possible to meet all the 
conditioiis of comprehensiveness > 

2. Disjointed incremental: a decision model appropriate for incremental decision 

settings which assumes that the decision maker wants to bring about 
smaller changes slightly different from the status quo and that he has 
little infor'^=^tion concerning how to achieve the change. The focus is 
more on current needs and problems and less on ultimate goals, 

3. Planned change: a decision model, appropriate for neomobilistic decision set- 

tings which has been conceptualized for situations calling for large cliange 
and in which there is Utile relevant information as to how change c^m be 
effectr^d. Tliis model involves many steps and agencies over a long span 
of time. 



Validators may find it useful, therefore, to categorize the practice under review in 
terms of 1) the appropriate decision-making setting; 2) the community's consciousness of 
the magnitude of the change; the appropriate decision-making model, and, as appropriate the 
relevant evaluation fo? mula. Where the exemplaiy p7.'actice resulted from both effective 
planning and high commu.iity concern there will undoubtedly be a high and quantifiable success 
factor* In short, the success of an exemplary educational practice is 



appropriate decision- 
making setting \ / effective planning 



congruent decision- / \ commimity concern 
making model 



S/R 
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Determining ''developmental costs" in response to the Cost Effectiveness question #3 
(p. 74) 

1. Determine number of months required for development 

2. Determine number of pupils for the total number of developmental months 

3. Determine total expenditures for the developmental time per.tod 
Example: 

Practice Qi) required 30 months of developmental time; 
involved 200 participants per year for 3 years, and 
required a total 3 year developmental expenditure of ' 
$120, 000, therefore, 

$200/pupil developmenta l costs 

600 (pupils) $120,000 



\ 
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Determi.nirxg the Extent of the Analysis of Data 



Page 64, question #11 



How extensively were the collected data analyzed, i*e, , 
did the project staff use a wide range of appropriate de^ 
criptive, infe'rerktial, and causal comparative analysis 
techniques ? 



Descriptive Statistics 



The descriptive statistical approach makes use of all the data concerning a population 
(the aggregate of all the cases that conform to some designated set of specifications). For 
instance, a computed mean of reading scores for a class of 22 students (the population) 
would be a descriptive statement about that population. 

Inferential Stati5=ties 



Inferential statistics builds on descriptive statistics m The purpose of inferential statis- 
tics is to surmise the properties of a population from a knowledge of the properties of only 
a sample of the population* 

Causal/Comparative Analysis 



From a knowledge of both descriptive and infereiitial (sometimes called inductive) 
statistics, three major types of evidence are necessary'' when comparing variables and test- 
ing for a causal/effect relationship between them. 

1. Evidence of Concomitant Variation, i.e. , that X (the independent or causal 
variable) and Y (the dependent variable) are associated in a way predicted 
by some hypothesis, either explicit or implicit. 

2. Evidence that Y did not occur before X. 

3. Evidence ruling out other factors as possible determining conditions of Y, 
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Determining Significant Improvement 



Page 55, question #14 



To what extent does the project evaluation contain ac- 
ceptable evidence that the performance of the participants 
was significantly improved? 



Note: '^significance" is being used as a judgment of value, and not in ti statistical sense* 
The validator's weighting of this question depends solelv on: 

a) his responses to questions 2 and 3 on page 60, and 

b) his analysis of the baseline data in question 1, columns 3 and 4 

If acceptable number weights were awarded these questions, and the participants did, in 
fact, reach the specified performance levels, then there was significant gain. The jvidgment, 
therefore, is not a matter of degree, but of whether or not the stated performance levels 
were achieved. 
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